System of vehicles equipped with imaging equipment for high-definition near real-time map generation

ABSTRACT

Described are street level intelligence platforms, systems, and methods that can include a fleet of swarm vehicles having imaging devices. Images captured by the imaging devices can be used to produce and/or be integrated into maps of the area to produce high-definition maps in near real-time. Such maps may provide enhanced street level intelligence useful for fleet management, navigation, traffic monitoring, and/or so forth.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.15/995,041, filed May 31, 2018, which is a non-provisional of, andclaims the benefit of priority of U.S. Provisional Patent ApplicationNo. 62/513,056, filed May 31, 2017, entitled “Near Real-Time StreetLevel Intelligence Platform,” the disclosure of each of which is herebyincorporated by reference in its entirety

BACKGROUND OF THE INVENTION

Street level intelligence has applications in a wide variety of fieldsincluding autonomous driving, architecture, construction, engineering,real estate, advertising, city planning, research and others. A fleet ofvehicles equipped with imaging equipment can produce and/or supplement amap with high-definition and/or near-real time data to provide enhancedstreet level intelligence. In general, the street level intelligence maycomprise telematics systems on which commercial fleet managers may relyto keep track of vehicles and drivers when they are out in the field.Some fleets may incorporate cameras to provide liability protection fromaccidents, moving violations or parking violations.

SUMMARY OF THE INVENTION

Prior methods for capturing street level intelligence at city scale areeither expensive or ineffective. Systems like Google Street View involveexpensive equipment mounted on modified vehicles, which are driven forthe express purpose of collecting data for creating maps and a visualdatabase. Solutions that use volunteers to “crowd source” the data neverreach useful scale. In both cases, the data collected becomes stalequickly.

In one aspect, disclosed herein are street level intelligence platformscomprising: at least one mapper vehicle, each mapper vehicle comprisingan active data capture system comprising a location determinationdevice, a LiDAR device, and at least one imaging device configured toactively capture data pertaining to an environment surrounding themapper vehicle within a territory; a fleet of swarm vehicles, each swarmvehicle comprising a passive data capture system comprising a locationdetermination device and a plurality of imaging devices configured topassively capture images of an environment surrounding the swarm vehiclewithin the territory; and a computing system comprising at least oneprocessor and instructions that when executed by the at least oneprocessor cause the at least one processor to create a street levelintelligence application comprising: a data processing pipelineconfigured to: receive data from the active data capture system, combinedata from the LiDAR, the imaging device, and the location determinationdevice, perform feature extraction on the combined data from the activedata capture system; receive data from the passive data capture systems,combine data from the imaging devices and the location determinationdevices, perform feature extraction on the combined data from thepassive data capture system; and merge the combined data from the activedata capture system with the combined data from the passive data capturesystem based on the extracted features; a street level intelligenceinterface; and a fleet manager interface. In various embodiments, the atleast one mapper vehicle comprises at least 2, at least 5, at least 10,at least 50, or at least 100 mapper vehicles. In some embodiments, theactive data capture system comprises at least one color panoramiccamera. In some embodiments, the at least one mapper vehicle refreshesdata at least bi-yearly. In some embodiments, the at least one mapper isa dedicated vehicle. In various embodiments, the fleet of swarm vehiclescomprises at least 10, at least 50, at least 100, at least 1000, or atleast 5000 swarm vehicles. In some embodiments, each swarm vehicle inthe fleet of swarm vehicles refreshes data at least monthly. In someembodiments, the fleet of swarm vehicles comprises a third-party fleetof vehicles to which passive data capture systems are affixed. In someembodiments, the passive data capture systems are affixed magnetically.In some embodiments, wherein the passive data capture system comprises aplurality of externally-powered smartphones. In further embodiments, atleast one smartphone faces outward from the front of the vehicle tocapture video and at least one smartphone faces outward from each sideof the vehicle to capture static images. In various embodiments, theterritory has an average radius of less than 10 miles, less than 50miles, or less than 100 miles. In some embodiments, the territory is aneighborhood, a city, a country, or a state. In some embodiments, themerging comprises: performing feature extraction on mapper data todetect landmarks; performing feature extraction on swarm data to detectlandmarks; comparing swarm landmarks within a predefine threshold ofeach mapper landmark to find the same landmarks; and updating the swarmlocation data when matching features are found. In some embodiments, thestreet level intelligence interface comprises one or more of: a mapoverlaid with defined street segments for which pertinent data has beencollected; tools allowing a user to view one or more of the streetsegments; tools allowing the user to select one or more of the streetsegments and add them to a cart; tools allowing the user to pay foraccess to the one or more street segments added to the cart; toolsallowing the user to download a payload for street segments, for whichaccess has been paid for, containing either 3D or 2D site intelligenceor pedestrian analytics; tools allowing the user access payloadspreviously paid for; and tools allowing the user share accessed payloadswith partners or other customers. In some embodiments, the fleet managerinterface comprises one or more of: tools allowing a user to see thecurrent location of each fleet vehicle; tools allowing the user todownload video offloaded from a fleet vehicle; tools allowing the userrequest priority offload of video from a fleet vehicle; and toolsallowing the user view driver quality data.

In another aspect, disclosed herein are computer-implemented systemscomprising: a digital processing device comprising: at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create a street levelintelligence application comprising: a software module for receivingdata from at least one active data capture system associated with amapper vehicle, each active data capture system comprising a locationdetermination device, a LiDAR device, and at least one imaging deviceconfigured to actively capture data pertaining to an environmentsurrounding the mapper vehicle within a territory; a software module forcombining data from the LiDAR, the imaging device, and the locationdetermination device; a software module for performing featureextraction on the combined data from the active data capture system; asoftware module for receiving data from a plurality of passive datacapture systems associated with a fleet of swarm vehicles, each passivedata capture system comprising a location determination device and aplurality of imaging devices configured to passively capture images ofan environment surrounding a swarm vehicle within the territory; asoftware module for combining data from the imaging devices and thelocation determination devices; a software module for performing featureextraction on the combined data from the passive data capture system; asoftware module for merging the combined data from the active datacapture system with the combined data from the passive data capturesystem based on the extracted features; a software module for providinga street level intelligence interface; and a software module forproviding a fleet manager interface.

In yet another aspect, disclosed herein are non-transitorycomputer-readable storage media encoded with a computer programincluding instructions executable by at least one processor to create anapplication comprising: a software module for receiving data from atleast one active data capture system associated with a mapper vehicle,each active data capture system comprising a location determinationdevice, a LiDAR device, and at least one imaging device configured toactively capture data pertaining to an environment surrounding themapper vehicle within a territory; a software module for combining datafrom the LiDAR, the imaging device, and the location determinationdevice; a software module for performing feature extraction on thecombined data from the active data capture system; a software module forreceiving data from a plurality of passive data capture systemsassociated with a fleet of swarm vehicles, each passive data capturesystem comprising a location determination device and a plurality ofimaging devices configured to passively capture images of an environmentsurrounding a swarm vehicle within the territory; a software module forcombining data from the imaging devices and the location determinationdevices; a software module for performing feature extraction on thecombined data from the passive data capture system; a software modulefor merging the combined data from the active data capture system withthe combined data from the passive data capture system based on theextracted features; a software module for providing a street levelintelligence interface; and a software module for providing a fleetmanager interface.

In yet another aspect, disclosed herein are computer-implemented methodsof generating real-time street level intelligence comprising: receivingdata from at least one active data capture system associated with amapper vehicle, each active data capture system comprising a locationdetermination device, a LiDAR device, and at least one imaging deviceconfigured to actively capture data pertaining to an environmentsurrounding the mapper vehicle within a territory; combining data fromthe LiDAR, the imaging device, and the location determination device;performing feature extraction on the combined data from the active datacapture system; receiving data from a plurality of passive data capturesystems associated with a fleet of swarm vehicles, each passive datacapture system comprising a location determination device and aplurality of imaging devices configured to passively capture images ofan environment surrounding a swarm vehicle within the territory;combining data from the imaging devices and the location determinationdevices; performing feature extraction on the combined data from thepassive data capture system; merging the combined data from the activedata capture system with the combined data from the passive data capturesystem based on the extracted features; providing a street levelintelligence interface; and providing a fleet manager interface.

BRIEF DESCRIPTION OF THE DRAWINGS

An understanding of the features and advantages of the described subjectmatter will be obtained by reference to the following detaileddescription that sets forth illustrative embodiments and theaccompanying drawings of which:

FIG. 1 shows an exemplary overview of a street level intelligence systemincluding a Mapper system and a Swarm system in accordance with someembodiments of the present disclosure;

FIG. 2 shows a front left view of a Swarm capture device with a housingin accordance with some embodiments of the present disclosure;

FIG. 3 shows a front right view of a Swarm capture device with a housingin accordance with some embodiments of the present disclosure;

FIG. 4 shows a bottom view of a Swarm capture device with a housing inaccordance with some embodiments of the present disclosure;

FIG. 5 shows a non-limiting example of a wiring diagram of Swarm capturedevices deployed in a vehicle in accordance with some embodiments of thepresent disclosure;

FIG. 6 shows a non-limiting example of Swarm selective capture logic inaccordance with some embodiments of the present disclosure;

FIG. 7 shows a non-limiting example of Swarm download logic inaccordance with some embodiments of the present disclosure;

FIG. 8 is a flow chart of a method of processing video to identifyhigh-priority events, according to an embodiment.

FIG. 9 is a screenshot of a Real-Time Events Service UI, according to anembodiment.

FIG. 10 shows a non-limiting example of a processing pipeline for aSwarm system, according to an embodiment;

FIG. 11 shows a non-limiting example of a wiring diagram of a Mappersystem in accordance with some embodiments of the present disclosure;

FIG. 12 shows a non-limiting example of a processing pipeline for amapper system, according to an embodiment;

FIG. 13 shows a non-limiting example of a Graphic User Interface (GUI)for a fleet manager in accordance with some embodiments of the presentdisclosure;

FIG. 14 shows a non-limiting example of a GUI for a street levelintelligence platform including a selection tool to examine pedestriancounts, according to an embodiment;

FIG. 15 shows an example of a pedestrian report for the area as selectedaccording to the GUI of FIG. 14 ; and

FIG. 16 shows a non-limiting example of a GUI for a street levelintelligence platform in accordance with some embodiments of the presentdisclosure.

DETAILED DESCRIPTION OF THE INVENTION

Described herein, in certain embodiments, are street level intelligenceplatforms comprising: at least one mapper vehicle, each mapper vehiclecomprising an active data capture system comprising a locationdetermination device, a LiDAR device, and at least one imaging deviceconfigured to actively capture data pertaining to an environmentsurrounding the mapper vehicle within a territory; a fleet of swarmvehicles, each swarm vehicle comprising a passive data capture systemcomprising a location determination device and a plurality of imagingdevices configured to passively capture images of an environmentsurrounding the swarm vehicle within the territory; and a computingsystem comprising at least one processor and instructions that whenexecuted by the at least one processor cause the at least one processorto create a street level intelligence application comprising: a dataprocessing pipeline configured to: receive data from the active datacapture system, combine data from the LiDAR, the imaging device, and thelocation determination device, perform feature extraction on thecombined data from the active data capture system; receive data from thepassive data capture systems, combine data from the imaging devices andthe location determination devices, perform feature extraction on thecombined data from the passive data capture system; and merge thecombined data from the active data capture system with the combined datafrom the passive data capture system based on the extracted features; astreet level intelligence interface; and a fleet manager interface.

Also described herein, in certain embodiments, are computer-implementedsystems comprising: a digital processing device comprising: at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create a street levelintelligence application comprising: a software module for receivingdata from at least one active data capture system associated with amapper vehicle, each active data capture system comprising a locationdetermination device, a LiDAR device, and at least one imaging deviceconfigured to actively capture data pertaining to an environmentsurrounding the mapper vehicle within a territory; a software module forcombining data from the LiDAR, the imaging device, and the locationdetermination device; a software module for performing featureextraction on the combined data from the active data capture system; asoftware module for receiving data from a plurality of passive datacapture systems associated with a fleet of swarm vehicles, each passivedata capture system comprising a location determination device and aplurality of imaging devices configured to passively capture images ofan environment surrounding a swarm vehicle within the territory; asoftware module for combining data from the imaging devices and thelocation determination devices; a software module for performing featureextraction on the combined data from the passive data capture system; asoftware module for merging the combined data from the active datacapture system with the combined data from the passive data capturesystem based on the extracted features; a software module for providinga street level intelligence interface; and a software module forproviding a fleet manager interface.

Some embodiments described herein relate to a software-implementedmethod (e.g., a non-transitory processor readable medium storing codeconfigured to be executed by a processor to perform a method) thatincludes capturing a video of a streetscape from a camera of asmartphone. A first pass of computer vision analysis can be performed onthe video of the streetscape to identify candidate high-priority events.A second pass of computer vision analysis can be performed on candidatehigh-priority events to identify a high-priority event. In this way, amore detailed analysis of the video of the streetscape can be performedthan would otherwise be possible with available computational resources.For example, available resources may be unable to process the video ofthe streetscape in real time to identify high-priority events, but maybe adequate to identify candidate high-priority events in real time andscreen candidate high profile events to identify (actual) high profileevents. An indication of high-profile events can be sent to a remoteanalysis service such that high-priority events can be integrated into amap of the streetscape. The remote analysis service may have greatercomputational resources than are available at a vehicle associated withthe smartphone.

Also described herein, in certain embodiments, are non-transitorycomputer-readable storage media encoded with a computer programincluding instructions executable by at least one processor to create anapplication comprising: a software module for receiving data from atleast one active data capture system associated with a mapper vehicle,each active data capture system comprising a location determinationdevice, a LiDAR device, and at least one imaging device configured toactively capture data pertaining to an environment surrounding themapper vehicle within a territory; a software module for combining datafrom the LiDAR, the imaging device, and the location determinationdevice; a software module for performing feature extraction on thecombined data from the active data capture system; a software module forreceiving data from a plurality of passive data capture systemsassociated with a fleet of swarm vehicles, each passive data capturesystem comprising a location determination device and a plurality ofimaging devices configured to passively capture images of an environmentsurrounding a swarm vehicle within the territory; a software module forcombining data from the imaging devices and the location determinationdevices; a software module for performing feature extraction on thecombined data from the passive data capture system; a software modulefor merging the combined data from the active data capture system withthe combined data from the passive data capture system based on theextracted features; a software module for providing a street levelintelligence interface; and a software module for providing a fleetmanager interface.

Also described herein, in certain embodiments, are computer-implementedmethods of generating real-time street level intelligence comprising:receiving data from at least one active data capture system associatedwith a mapper vehicle, each active data capture system comprising alocation determination device, a LiDAR device, and at least one imagingdevice configured to actively capture data pertaining to an environmentsurrounding the mapper vehicle within a territory; combining data fromthe LiDAR, the imaging device, and the location determination device;performing feature extraction on the combined data from the active datacapture system; receiving data from a plurality of passive data capturesystems associated with a fleet of swarm vehicles, each passive datacapture system comprising a location determination device and aplurality of imaging devices configured to passively capture images ofan environment surrounding a swarm vehicle within the territory;combining data from the imaging devices and the location determinationdevices; performing feature extraction on the combined data from thepassive data capture system; merging the combined data from the activedata capture system with the combined data from the passive data capturesystem based on the extracted features; providing a street levelintelligence interface; and providing a fleet manager interface.

Some embodiments described herein relate to a system that includesmultiple vehicles each equipped with a vehicle-mounted smartphone and avideo-and-map-integration device. Vehicle-mounted smartphones cancontinuously capture streetscape video. The vehicle-mounted smartphonesmay be unable to transfer raw streetscape video to the video-and-mapintegration device, for example, because average (e.g., daily) bandwidthfor a cellular data network is less than what would be required totransfer video. The vehicle-mounted smartphones can therefore locallystore streetscape video (e.g., using internal memory and/or peripherallocal storage). In some instances, vehicle-mounted smartphones can beoperable to store at least eight hours of streetscape footage.Vehicle-mounted smartphones can also be operable to identifyhigh-priority features in streetscape video and send a portion of thevideo containing the high-priority feature to thevideo-and-map-integration device, e.g., via the smartphone's cellulardata radio and/or a cellular data network. The video-and-map integrationdevice can be operable to integrate high-priority events into a map andsend updated maps to the fleet of vehicle-mounted smartphones over thecellular data network. In this way, the fleet can be alerted tohigh-priority events in near real-time (e.g., within 3 hours). Whenvehicles returns to a home location (e.g., a garage), vehicle-mountedsmartphones can connect to a WiFi network, which may have higherbandwidth than the cellular data network and/or at which the vehiclesmay spend a greater amount of time, and transfer locally storedstreetscape video to the video-and-map-integration device via thevehicle-mounted smartphone's WiFi radios. The video-and-map-integrationdevice can be operable to integrate video received via WiFi into the mapso that the map is updated at least daily. Updated maps can be sent tomap-viewer devices, vehicle-mounted smartphones, and/or navigationdevices.

Certain Definitions

Unless otherwise defined, all technical terms used herein have the samemeaning as commonly understood by one of ordinary skill in the art towhich this invention belongs. As used in this specification and theappended claims, the singular forms “a,” “an,” and “the” include pluralreferences unless the context clearly dictates otherwise. Any referenceto “or” herein is intended to encompass “and/or” unless otherwisestated.

As used herein, “Swarm” refers to a passive capture system capable ofpassively capturing images and/or videos of a surrounding environmentand depicts traffic impacting events in real-time. In some instances, aSwarm system can be provided by a third-party fleet which may havemultiple vehicles outfitted with Swarm systems for data collecting andprocessing. Intelligence products, such as pedestrian foot traffic foreach segment of the map, can be derived at the block or neighborhoodlevel with a relatively short refresh rate, for example, nightly,weekly, or a monthly.

As used herein, “Mapper” refers to an active capture system capable ofactively capturing images and videos of a surrounding environment andgenerating high-resolution graphical representations, such as a 2D orthree-dimensional (3D) base map. In some instances, the Mapper systemcan be arranged on a dedicated vehicle with multiple sensors for sensingthe surrounding environment and collecting data. The data can be usedfor deriving the high-resolution 2D and 3D map at an address level witha relatively long refresh rate, for example, a bi-yearly refresh rate.

In accordance with some embodiments of the present disclosure, theMapper system and Swarm system as discussed above can be configured tooperate in a combined manner, thereby dataset provided by the Mappersystem being more accurate and dataset provided by the Swarm systembeing updated more regularly. In some instances, the combination ofthese two types of datasets can be carried out via feature extractionand feature matching based on machine learning.

Overview

Referring to FIG. 1 , in a particular embodiment, a street levelintelligence system 100 may comprise two sub-systems, i.e., a Mappersystem and a Swarm system. The Swarm system as shown may comprise aCollect module 101, a Fuse module 102, and an Analyze module 103.Similarly and correspondingly, the Mapper system as shown may comprise aCollect module 104, a Fuse module 105, and an Analyze module 106.Further illustrated are a Merge module 107 and a Transform module 108,which may be shared by the Mapper system and Swarm system and arecapable of merging the data from the analyze modules and transformingthe merged data into a desired format for utilization.

In some instances, the Collect module 101 of the Swarm system maycomprise one or more imaging devices for capturing still images orvideo. The imaging device can be configured to detect electromagneticradiation (e.g., visible, infrared, and/or ultraviolet light, etc.) andgenerate image data based on the detected electromagnetic radiation. Theimaging device may include a charge-coupled device (CCD) sensor or acomplementary metal-oxide-semiconductor (CMOS) sensor that generateselectrical signals in response to wavelengths of light. The resultantelectrical signals can be processed to produce image data. The imagedata generated by the imaging device can include one or more images,which may be static images (e.g., photographs), dynamic images (e.g.,video), or suitable combinations thereof. The image data can bepolychromatic (e.g., RGB, CMYK, HSV, etc.) or monochromatic (e.g.,grayscale, black-and-white, sepia, etc.). The imaging device may includea lens configured to direct light onto an image sensor.

In some embodiments, the imaging device can be embodied as a smartphonewhich may include one or more cameras capable of capturing still ordynamic image data (e.g., video). The camera can be a still camera thatcaptures static images (e.g., photographs). The camera may capture bothdynamic image and static images. The camera may switch between capturingdynamic images and static images. Although certain embodiments providedherein are described in the context of smartphones, it shall beunderstood that the present disclosure can be applied to any suitableimaging device, and any description herein relating to a cameras canalso be applied to any suitable imaging device, and any descriptionherein relating to cameras can also be applied to other types of imagingdevices. A camera can be used to generate 2D images of a 3D scene (e.g.,an environment, one or more objects, etc.). The images generated by thecamera can represent the projection of the 3D scene onto a 2D imageplane. Accordingly, each point in the 2D image corresponds to a 3Dspatial coordinate in the scene. The camera may comprise opticalelements (e.g., lens, mirrors, filters, etc.). The camera may capturecolor images, greyscale image, infrared images, and the like. The cameramay be a thermal imaging device when it is configured to captureinfrared images.

In some embodiments, the imaging device (e.g., a smartphone) and/orother local (e.g., on-vehicle) computational resources may be limited.For example, the imaging device may include moderate processing and/ormemory (e.g., partially or completely disposed within the housing of thesmartphone). Furthermore, the imaging device, when in the field, mayhave a limited bandwidth, including periods of no connectivity to morerobust processing capabilities, such as cloud computing resources. Forexample, a cellular data network accessible to the imaging device maynot provide sufficient bandwidth to transfer continuously collectedvideo and/or rates charged by cellular data network carriers fortransferring data may make transfer of continuously collected videocost-prohibitive. Accordingly, as described in further detail here, someembodiments relate to systems and methods for processing and analyzingdata captured by the imaging device to provide better results than wouldotherwise be possible with available computational resources and/or formanaging data transfers from the imaging device to remote storage and/oranalysis facilities.

The Collect module 101 of the Swarm system can be started automaticallywhen the Swarm system boots. In some instances, when the Swarm system isinstalled on a smartphone which may be used as an imaging device fordata collection, the collect module 101 can be configured to commencecapturing the images or videos once the smartphone is disconnected froma Wi-Fi network, for example, when the smartphone is mounted on avehicle of the fleet and the vehicle is moving out of the fleet garageand in the street. During the capture processing, if high-resolutionstill images are captured, motion blur may be encountered and should betaken into account through a Swarm selective capture logic asillustrated in FIG. 6 , as is discussed herein.

In addition to using the imaging device to capture the images andvideos, the Collect module 101 may also comprise one or more sensors,for example, vision sensors, positioning sensors such as Global PositionSystem (GPS) sensors, or inertial sensors such as accelerometers,gyroscopes, and/or gravity detection sensors. Thereby, additional datasuch as time, frame numbers, locations, speeds, accelerations, lightlevels and counts nearby of Bluetooth/Wi-Fi devices can be captured orcollected by one or more of these sensors and then can be fused togetherwith the image or video data by the Fuse module 102 of the Swarm system.Upon fusing operations via the Fuse module 102, real-time or nearreal-time imagery with approximate or changing location can be achievedfor Video-on-Demand (VOD) services, for example, requested by a fleetmanager or a potential third-party for driver analysis.

In some instances, the Collect module 101 of the Swarm system mayadditionally transmit location and status reports to a remote server ona regular basis, for example, via a Hyper Text Transfer Protocol (HTTPS)over a wireless communication system. The remote server herein may be acloud server at a remote site or a server located or owned by a thirdparty.

In some instances, the Analyze module 103 of the Swarm system mayanalyze the captured imagery and apply processing algorithms todetermine the pedestrian traffic, which will be discussed in detaillater with reference to FIG. 10 .

In some instances, for collecting and obtaining high-resolution imagesand videos, the Collect module 104 of the Mapper system may comprisehardware with high precision and definition. For example, the Collectmodule 104 of the Mapper system may comprise an advanced GPS/GlobalNavigation Satellite System (GNSS), or Inertial Measurement Unit (IMU)sensors for location and orientation determination. Further, the Collectmodule 104 may comprise one or more LiDAR sensors for forming pointcloud data and a spherical digital video camera system, such asPointGray Ladybug 5, for filming or capturing panoramic images. Thehardware mentioned herein is only for an illustrative purpose and anexample of the Mapper system including suitable hardware is describedwith reference to FIG. 10 .

The various types of data, such as high-resolution images and videos,time data, location data can be collected into files with correspondingformats. The files may include, but are not limited to, WMV files, ASFfiles, ASX files, RM files, RMVB files, MPG files, MPEG files, MPEfiles, 3GP files, MOV files, MP4 files, M4V files, AVI files, DAT files,MKV files, FLU files, VOB files, JPG files, TIFF files, RAM files, BMPfiles, GIF files, PNG files, PCX files, WMF files, PCX files, SWF files,GDF files, KIWI files, NavTech files, raw network PACP files, LadybugPGR files, and GPS files. In some embodiments, some of these files arefused with one another as appropriate by the Fuse module 105 of theMapper system. For example, the LiDAR data from PCAP files can becombined with GNSS data to create point cloud slices in CSV files andposition/orientation information in POSE files. By a further combinationof SCAN, POSE, and rectified JPG files, registered and colorized pointcloud slices can be created. After that, the Analyze module 106 cangenerate high definition or resolution map and site intelligence can beachieved by performing feature extraction and other necessary dataprocessing. It should be understood that the descriptions of theoperations of the Fuse module 105 and the Analyze module 106 are onlyillustrative of some aspects of the processing performed thereby anddetailed discussions will be made later with reference to FIGS. 10 and12 , which illustrate further details of the processing flow of theMapper system and Swarm systems.

As shown in FIG. 1 , subsequent to analyzing the data respectively bythe Analyze module 103 and Analyze module 106, the resultant data fromthe Mapper system and Swarm system can be merged and transformed by theMerge module 107 and Transform module 108, respectively, in accordancewith some embodiments of the present disclosure. In some instances, withthe aid of a deep learning technique, one or more machine learningmodels can be established and trained to detect objects or features,such as windows, signage (e.g., road signs or advertising signs) andother landmarks. For example, when the features are related to thewindows, feature extraction can be performed on the Mapper data todetect windows and likewise, feature extraction can be performed on theSwarm data to detect windows. Then, the windows detected in the Mapperdata can be compared to those detected in the Swarm data so as todetermine whether they are the same. If this is the case, it means thatthe Swarm data is correct in this regard. However, if this is not thecase, then the Swarm data with respect to the windows can be updated,for example, location data, thereby improving the accuracy of the Swarmdata. Further, in some instances, with the aid of the Merge module 107,the image can be localized (i.e., image localization) and thereforereal-time high definition map can be obtained at the Transform module108. In addition, based on the merging operations at the Merge module107, predictions regarding the pedestrian foot traffic can be changedand mapping can be prioritized at the Transform module 108.

The above has discussed modules or blocks of the street levelintelligence system and main functionality thereof. It should beunderstood that the descriptions herein of these modules and theiroperations are only for illustrative purposes and a person skilled inthe art, based on the teaching of the present disclosure, can add,remove, omit, or combine one or more of the modules without departingfrom the spirit and scope of the present disclosure. Further, by anefficient and effective combination of Mapper data from the Mappersystem and Swarm data from the Swarm system, the street levelintelligence system in accordance with the embodiments of the presentdisclosure can be both high-resolution and kept current and insights canbe derived from two combined datasets using machine learning technique.Further, with the aid of the machine learning based feature extractionand feature matching, the Swarm data can be made more accurate and theMapper data can be updated more regularly.

Referring to FIG. 2 , in a particular embodiment, a Swarm capture device200 may comprise a housing for accommodating an imaging device. Thehousing may have two parts, i.e., a first main component 201 and asecond main component 202, which can be designed as complementary forfastening and tightening the imaging device in a vehicle. In someinstances, the first main component and the second main component can bemovably connected to one another with a connecting mechanism, such as alocking or clamping mechanism. In some instances, the first maincomponent and the second main component can be a single piece with apull-out design such that the imaging device can be inserted thereinwhen the two main components are pulled in different directions. Thehousing may further comprise two protrusion components 204 and 205,which are complementary to one another and arranged respectively on thefirst main component and second main component. Both protrusioncomponents can be attached to one another by, for example, a connectingcomponent, such as a bolt, a belt, a screw, a nut or the like, such thatthe imaging device can be tightly clasped in the housing. Further, theprotrusion components herein can also be attached to other connectingmechanisms or components in the vehicle such that the imaging device canbe fastened to the vehicle for image collection.

The imaging device 203 as shown is embodied as a smartphone with acamera 206 for exemplary purposes; other imaging device, such as acamera, is also possible. The smartphone can be placed or inserted inthe housing for image capture. In some instances, the Swarm capturedevice may optionally have one or more power units, such as batteryunits for powering the smartphone, thereby extending the shooting timeof the smartphone when the power level of the smartphone is lower than apredetermined threshold. Further, the Swarm capture device may furthercomprise one or more slots or ports for connecting the smartphone toother devices, for example, other imaging devices arranged on the samevehicle for imaging capture, which is exemplarily shown in FIG. 5 .

In some embodiments, some functions of the smartphone can be disabledand thereby it is dedicated to capture the image and video surroundingthe vehicle. For example, when placed in the housing and prepared forshooting, the call function of the smartphone can be disabled. In someinstances, to save the power, other functions, such as receiving shortmessages, can also be disabled. Further, the smartphone can be set intoa flight mode in which no communication can be made and thus power ofthe smartphone can be further saved for environmental data collection.

Referring to FIG. 3 , a front right view of a Swarm capture device 300is shown. It should be understood that the Swarm capture device 300 canbe similar or equal to the Swarm capture device 200 as shown in FIG. 2 .Therefore, any descriptions of the Swarm capture device with respect toFIG. 2 can be equally applied to the Swarm capture device 300 herein.

As shown in FIG. 3 , the Swarm capture device 300 has a housingcomprising a first main component 301 and a second main component 302which can accommodate an imaging device 303 inserted therein. Further,the housing may further comprise two protrusion components for fasteningthe Swarm capture device 300 in place when capturing the images. Theimaging device 303 herein is depicted as a smartphone, which can be thesame as shown in FIG. 2 . Similar to the descriptions made with respectto FIG. 2 , the smartphone herein can be configured to be exclusivelyused for image collection. Additionally or alternatively, the smartphonecan also be connected with a Bluetooth connection; therefore, a callfunction can still be enabled even if the smartphone is operating tocapture the images and videos.

The smartphone as shown in FIGS. 2 and 3 can be configured to run on anysuitable operation system (OS), for example, an Android OS or a Mac OS.In some instances, the smartphones in accordance with some embodimentsof the present disclosure can be configured to run a customized versionof a LinageOS (formerly Cyanogen) Android distribution. Thiscustomization may enable the smartphone of boot-on-power, remote access,and additional logging. For example, as discussed in further detailherein, a smartphone can be connected to vehicle power in such a mannerthat the smartphone can detect when the vehicle engine has been turnedon and/or off. The smartphones can be configured to operate in a“headless” manner, for example, disabling call functionality, someand/or all input modes, configured to provide custom output, and/or soforth. For example, the smartphones can be configured to power on whenthe vehicle is turned on, automatically enter a data capture mode (e.g.,capturing video and/or pictures of a streetscape), disable callfunctionality, and/or power off or enter a power saving state when thevehicle is turned off. Moreover, the smartphones can be configured toautomatically capture data when certain conditions are met, such as whenthe smartphone detects it has left a home base or garage (e.g., via GPSand/or inertial sensors), when it is in an area that has not previouslyand/or recently been imaged (e.g., based on data received from a remoteanalysis, monitoring, and/or coordination device (e.g., a Real-TimeEvents Service, as described in further detail herein). The smartphonescan also be configured to automatically cease capturing data whencertain such as when the smartphone detects it has entered a home baseor garage. Furthermore, the smartphones can be configured toautomatically send data to a remote analysis, monitoring, and/orcoordination device and/or manage modes through with data is sent. Forexample, smartphones can be configured to use a relatively low-bandwidthand/or high cost network (e.g., cellular data networks, such as an LTEnetwork) to send indications of high-priority events, while storinglow-priority data (e.g., video data) locally until a relativelyhigh-bandwidth and/or low cost network is available (such as WiFi or awired local area network). Similarly stated, the smartphones can beconfigured to connect to WiFi when they detect that they have entered ahome base or garage and upload video data via the WiFi network. Inaddition or alternatively, the smartphones can be configured toperiodically contacts a remote analysis, monitoring, and/or coordinationdevice for instructions, map updates, indications of high-priorityevents detected by other smartphones, and/or the like.

Referring to FIG. 4 , a bottom view of a Swarm capture device 400 isshown. The Swarm capture device 400 herein may be similar or equal tothe one respectively shown in FIGS. 2 and 3 . Therefore, anydescriptions of the Swarm capture device made before with respect toFIGS. 2 and 3 can be equally applied to the Swarm capture device 400.The Swarm capture device 400 has a housing comprising a first maincomponent 401 and a second main component 402. Also shown at the bottomof the Swarm capture device are two mounting components, such as a firstmagnetic component 403 and a second magnetic component 404. The firstmagnetic component 403 and second magnetic component 404 can providesufficient magnetism to fix the Swarm capture device 400 to a metallicsurface, for example, at the front sides, left and right sides of thevehicle. Therefore, it would be easy for the user to mount the Swarmcapture device on the vehicle or remove the Swarm capture device fromthe vehicle. It should be understood that magnetic components herein areonly for illustrative purposes and any other coupling mechanism, such asfasteners, actuation elements, joints, hinges, bolts, screws, etc. canalso be used for movably coupling the Swarm capture device to thevehicle. The vehicle herein may be an autonomous vehicle or unmannedvehicle.

Referring to FIG. 5 , a wiring diagram 500 of Swarm capture devicesdeployed in a vehicle 501 is shown. In a particular embodiment, fourSwarm capture devices 504, 505, 506, and 507, each of which is similaror equal to the one discussed with respect to FIGS. 2-4 are arranged onthe vehicle 501. As shown, the Swarm capture devices 505 and 507 arearranged at the top of the cab of the vehicle and the lens of thecameras are forward-facing. Further, the Swarm capture devices 504 and506 are arranged at the two sides of the cab and the lens of the camerasare sideways-facing. With the aid of this arrangement of the Swarmcapture devices, the view around the front part of the vehicle can becaptured. As discussed before, the Swarm capture devices can beconfigured to capture video or still images. In a typical configuration,two front-facing Swarm capture devices (e.g., 505 and 507) can capturevideo while two side facing Swarm capture devices (e.g., 504 and 506)can capture high-resolution still images.

It is to be understood that the arrangement of the Swarm capture devicesherein is only for exemplary purposes and a person skilled in the artcan envisage that the Swarm capture devices can be arranged at anywhereof the vehicle as appropriate. For example, the Swarm capture devicescan also be disposed at the rear of the vehicle, for example, near theback mirror.

For centralized control and data synchronization, a microcontroller 502can be optionally applied to control the Swarm capture devices, forexample, for powering the Swarm captured devices with. e.g., 5V power,and for synchronizing them via Universal Serial Bus (USB) ports. Tobetter power-up the Swarm capture devices, a DC-DC converter may bewired into a 12V power system of the vehicle using a fuse tap that ispowered on when the vehicle's motor is running and powered off when thevehicle's motor stops running. In this case, custom software can be usedfor allowing the Swarm capture devices to detect when the vehicle'smotor is running, power up and/or begin capturing when the vehicle isturned on and/or power down (e.g., enter a standby or low energy state)when the vehicle's motor is turned off. While the vehicle's motor isrunning, the Swarm capture device can receive power from the DC-DCconverter.

Referring to FIG. 6 , in a particular embodiment, a Swarm selectivecapture logic 600 is used for selectively taking and keeping an image,thereby ensuring the quality and effectiveness of the still images.

As shown in the flow chart of a process for taking a picture at theupper part of FIG. 6 , at block 601, the still capture logic candetermine whether the present time is daylight. If the present time isnot daylight but at night unsuitable for taking a picture, then thelogic may decide not to take a picture at block 606. However, if thepresent time is daylight suitable for taking a picture, then the flowproceeds to block 602, at which it is determined whether the currentlocation is near the garage where the vehicle is parked. If it isdetermined that the vehicle is near the garage, for example, the vehiclenot moving out of the garage, then the logic can decide not to take apicture at block 606. If the current location is not near the garage,for example, the vehicle moving out of the garage, then at block 603,the logic can further determine whether the vehicle travels equal to ormore than 3 meters. If this is the case, then the Swarm capture devicemay take a picture at block 605. Otherwise, the flow proceeds to block604, at which the logic can determine whether the last picture was taken10 second ago. If this is the case, then the Swarm capture device cantake a picture at block 605. If this is not the case, then the flowreturns back to block 606, at which it decides not to take a picture.This may be due to the fact that frequently taking still images is notdesirable in some situations.

It should be understood that the flow chart of the process forselectively taking a picture as discussed above is only for illustrativepurposes and a person skilled in the art can understand that otherjudging conditions, in addition to time or location, can also be appliedfor deciding whether or not to take a still image. For example, thejudging condition may be a power level or a storage level of the Swarmcapture device, a given time slot for taking pictures as specified bythe user. Additionally, in some instances, once the Swarm capture deviceis disconnected from a Wi-Fi connection and its GPS reports it has movedsufficiently far from the fleet garage, then imaging capture will begin.

As shown at a lower left side of FIG. 6 , a still picture is taken bythe Swarm capture device and divided into nine grids or regions, whereina middle grid and four corner grids are denoted by 3, 1, 2, 4, 5,respectively. Based on the still picture as pre-processed in this way, aprocess for deciding whether or not to keep the captured still picturemay be performed as illustrated in the flow chart at a lower right sideof FIG. 6 . At block 607, the Swarm selective capture logic maycalculate a Laplacian variance for each pixel in each of five regions 1,2, 3, 4, and 5. Then at block 608, the logic may determine whether theLaplacian variances calculated for all five regions are less than apredetermined threshold. If this is the case, then at block 609, thelogic may decide to keep the picture. Otherwise, if the Laplacianvariances calculated for all five regions are equal to or greater thanthe predetermined threshold, then the logic may decide to delete thecaptured picture at block 610. In this way, motion blur that may occurin the captured image can also be eliminated or alleviated. In aninstance where some regions have Laplacian variances greater than thepredetermined threshold, while other regions have Laplacian variancesbelow the predetermined threshold, the image may be kept. Alternatively,the image may be kept if the number of regions having Laplacianvariances greater than the predetermined threshold is less than anotherpredetermined threshold (e.g., if Laplacian variances for a majority ofregions is less the predetermined threshold). In this way, partiallyblurred images, such as may occur when imaging a streetscape having afast-moving object (e.g., a car) may be retained.

It should be noted that the process for deciding whether or not to keepthe picture as discussed above is only for illustrative purposes and aperson skilled in the art can envisage other solutions or alternativesfor deciding whether or not to keep the picture. For example, althoughthe picture is divided into nine grids and five of them are selected, itcan also be divided into more or less grids so as to meet the precisionrequirements. Further, although the Laplacian variance is used herein,other algorithms suitable for determining the quality of the image canalso be applied as envisaged by those skilled in the art.

In some instances, watchdog software can be installed on a Swarm capturedevice to monitor and ensure that the image and video capture logic (orsoftware) are working properly. In addition, the watchdog software mayalso be responsible for downloading and installing software or firmwareupdates. In some instances, in addition to taking pictures andcollecting image data, the Swarm capture device can be configured tocollect and generate other types of data, including but not limited totime, frame numbers, locations, accelerometer states, light levels, andcounts nearby of Bluetooth/Wi-Fi devices in a separate metadata file. Incase the microcontroller is being used, synchronization pulse data canalso be included in the metadata file.

Referring to FIG. 7 , in a particular embodiment, Swarm download logicis used for controlling video download, wherein a process 701 fordownloading recent video is shown in the flow chart at the upper part ofFIG. 7 and a process 708 for downloading historical video is shown inthe flow chart at the lower part of FIG. 7 .

First referring to the process 701, at step 702, a customer or a clientmay transmit to a fleet manager a request for video onboard the vehicle713 which can be autonomous vehicle in some instances. The request canbe transmitted via a smartphone of the user, for example, with the aidof a client application installed on the smartphone. Upon receipt of therequest by the fleet manager, for example, via a remote server, thefleet manager may forward the request at step 703 to a Command andControl Center (CNC) where the requests from different fleet managersmay be collected and processed. Then, the CNC may transmit the requestto the vehicle 713 via a wireless communication network, such as a LongTerm Evolution (LTE) system. Through the LTE network, the vehicle 713can transmit its status to the CNC at step 704 or poll the CNC forpotential requests. In case the vehicle 713 receives the request at step705, it can transmit or upload the video collected during the travel viathe LTE network to a cloud storage system such as the Amazon SimpleStorage Service (S3), whereby the requested or demanded video can betransmitted to the customer. In other words, the customer can downloadthe video from the database S3 and view it locally on his or hersmartphone.

It can be understood the above process can be performed when the vehicleis moving in the street or crossing blocks. Further, the steps as shownin the process 701 are only for exemplary purposes and a person skilledin the art can understand that some of the steps can be omitted orcombined as appropriate. For example, in some instances, the CNC and thefleet manager may be co-located and therefore the request can bedirectly handled by the CNC without any involvement of the fleetmanager. Additionally or alternatively, the CNC and database S3 may bealso co-located and therefore the CNC may directly forward the videofrom the database S3 to the client without establishing a new connectionbetween the database S3 and client. In some cases, the Swarm capturedevices can send location and status reports regularly via HTTPS overthe LTE network to the database S3 for records and statistical analysis.Alternatively, the Swarm capture devices can also send the location andstatus reports regularly or in real-time to the fleet manager such thatthe fleet manager can monitor the vehicle at any given time.

Now referring to the process 708, at step 702, when the vehicle 713 isdriven into the garage, a Wi-Fi connection can be established with theSwarm capture devices and capturing operations performed by the Swarmcapture devices may stop. Then, Swarm capture devices can offload allthe image data to the database S3. Once image data has been completelyoffloaded and the vehicle has been turned off, the Swarm capture deviceswill shut down immediately. Alternatively, the offloading operations canbe carried out at any suitable location upon request of the Swarmcapture devices or a remote server, such as the remote server controlledby the fleet manager. Additionally, the Swarm capture devices canrequest offloading operations to the remote server and can beginoffloading operations upon receipt of confirmation information from theremote server.

In some instances, when the user sends to the fleet manager a requestfor video at step 710, the fleet manager can forward the request to thedatabase S3 through the Wi-Fi network at step 712. Upon receipt of therequest from the fleet manager, the database S3 may transmit therequested video to the user at step 711. The user herein can be a personor a company, such as a third party (e.g., Zendrive), and therefore, therequested video including priority data can be used for driver analysis.

It can be understood that the process 708 can be performed when thevehicle returns to the garage or is parked in the garage. Further, thesteps as shown in the process 708 are only for exemplary purposes and aperson skilled in the art can understand that some of the steps can beomitted or combined as appropriate. For example, in some instances, thedatabase S3 and the fleet manager may be co-located and therefore therequest can be directly handled by the database S3 without anyinvolvement of the fleet manager. Additionally or alternatively, thefleet manager can communicate with the database S3 via a separateconnection, for example, via a LTE connection.

FIG. 8 is a flow chart of a method of processing video to identifyhigh-priority events, according to an embodiment. At least portions ofthe method shown and described with reference to FIG. 8 can be performedby computing entities with relatively low computing power, such as asmartphone having a Snapdragon 820 processor. Thus, the method describedwith reference to FIG. 8 allows relatively modest computing resources toproduce results that they would be unable to achieve using knowntechniques. For example, the method described with reference to FIG. 8can identify high priority events from real-time video using a processorincapable of performing known machine vision techniques on video in realtime.

A Swarm capture device can be operable to continuously record video. Thevideo may be of high quality (e.g., having a resolution of at least1280×720 and a frame rate of at least 30 frames/second). As discussed infurther detail herein, it may not be feasible to wireless transmitcontinuously captured video over cellular data networks. Therefore,captured video can be stored locally, for example on memory within asmartphone housing and/or on a local hard drive or flash memory device.In some embodiments, there can be sufficient local (e.g., on-vehicle)storage capacity to store at least 4 hours, at least 8 hours, at least12 hours, or at least 24 hours of video. In such an instance, locallystored video can be transferred to an analysis, monitoring, and/orcoordination device (e.g., a Real-Time Events Service, avideo-and-map-integration device, or other suitable device and/orservice) via a WiFi radio/network, for example when the vehicle returnsto a home base or garage. The analysis, monitoring, and/or coordinationdevice can integrate video received from the Swarm capture devices viaWiFi into the map such that the map is updated with video data receivedvia WiFi, at least every 4 hours, every 12 hours, daily, weekly,monthly, or at any other suitable time interval. Furthermore, theanalysis, monitoring, and/or coordination device can be operable send asignal representing a map updated with content received from the Swarmcapture devices to a map-viewer device (e.g., a personal computer,smartphone, tablet, etc. running a web browser or map-viewerapplication), a Swarm capture device, and/or a navigation device. Insome embodiments, the Swarm capture device can be communicativelycoupled to one or more sensors, such as GPS, LiDAR, other image capturedevices and/or so forth, and associate data received from such sensorswith captured video. For example, GPS coordinates, LiDAR point clouds,and/or so forth can be associated with captured video and/or appended asmetadata to the video.

The Swarm capture device can be operable to perform a first passcomputer vision analysis of the video in real time. The first pass canidentify candidate high-priority features. For example, a bandpassfiltering technique can be performed on the video to identify predefinedcharacteristic signals associated with high-priority features. Forexample, if a safety-orange or safety-yellow color is identified, theseframes can be identified as potentially containing traffic cones orbarrels. Similarly, if flashing blue or red lights are detected, thevideo can be identified as potentially containing emergency vehicles.The first pass can be performed in real-time, potentially at a lowerframe rate than the video is captured (e.g., at 1 frame/second, 10frames per second, 30 frames per second, etc.).

A second pass can of machine vision/machine learning analysis can beperformed on frames and/or video clips identified as candidatehigh-priority feature. The second pass can be more computationallyintensive, and the processor of the Swarm capture device may haveinsufficient resources to perform the analysis performed during thesecond pass in real time on the continuously recorded video. Byselectively performing the second pass on candidate high-priorityfeatures, Swarm capture device can perform higher-quality imagerecognition tasks than would otherwise be possible. The second pass can,for example, identify cones, jersey barriers, roadblocks and/or barrelsto identify a construction area. In addition or alternatively, thesecond pass can identify emergency light bars, police logos/markings,and police roadblocks. The second pass can be performed in near-realtime (e.g., within five minutes of the video being captured).

High-priority events identified during the second pass can be sent to aReal-Time Event Service for further processing and/or analysis. Forexample, a portion of the video captured by the Swarm capture device canbe uploaded with metadata (e.g., location data, time stamp, indicationsof high-priority features, etc.) to the Real-Time Event Service, forexample over a cellular data network (e.g., LTE). In some instances, theReal-Time Event Service may possess computational and/or human resourcesbeyond those available in the Swarm vehicle. Furthermore, the Swarmcapture device may be unable to send full raw video to the Real-TimeEvent Service. For example, the cellular data network may haveinsufficient average daily bandwidth to allow the Swarm capture deviceto upload continuously captured video. Similarly stated, the bandwidthat which video is captured by the Swarm capture device may, on average,exceed the bandwidth of a cellular data network available to the Swarmcapture device.

The Real-Time Event Service can perform additional machinevision/machine learning and/or human evaluation to determine the impactof the high-priority event. For example, the Real-Time Event Service canapply machine learning models to verify the detection of event objectsidentified in the second pass with additional accuracy. A notificationcan appear in a Real-Time Events Service UI, which can be used todetermine and/or record impact of the high-priority event. FIG. 9 is ascreenshot of a Real-Time Events Service UI, according to an embodiment.In some embodiments, a GIS analyst, using the Real-Time Events ServiceUI can watch video along with a synchronized map. The analyst candetermine if the high-priority event impacts traffic. A polygon can bedrawn (by the GIS analyst and/or machine learning processes) and thetype of impact elected (e.g., from a drop-down menu). Swarm vehiclesand/or subscribed parties can be notified of the possible event,confirmation status, and impacted area. Additionally or alternatively,Swarm capture devices can periodically poll a Real-Time Events API forupdates on high-priority events (e.g., occurring within a predeterminedrange of the Swarm vehicle) and/or the Real-Time Event Service can pushupdates on high-priority events to Swarm capture devices.

The Real-Time Events Service can be operable to determine when a highpriority event occurred. For example, upon receiving an indication of ahigh-priority event from a Swarm device, the Real-Time Events Servicedetermine if any other Swarm devices passed by the location of thehigh-priority event (e.g., within the last hour, last 6 hours, last 24hours, last 48 hours, last week, etc.) and request video footage of thelocation of the high-priority event from any such Swarm devices. TheReal-Time Event Service can analyze video received from such Swarmdevices and determine a window in which the high-priority eventappeared.

Swarm capture devices passing an identified high-priority events can beconfigured to identify the absence of the event (e.g., using a processsimilar to the two pass analysis described above). In the event is notdetected by a subsequent Swarm capture device, the Real-Time EventService can be configured to update the map to remove the high-priorityevent.

Referring to FIG. 11 , in a particular embodiment, a Mapper 800 inaccordance with some embodiments of the present disclosure can comprisean antennas module 801 (e.g., Novatel 702GG Antenna), a GPS/GNSS module802 (e.g., Novatel SPAN-IGM-S1 GNSS), a data collection module 803, aLiDAR module 804 (e.g., Velodyne VLP-16 “Puck”), a LiDAR interfacemodule 805 (e.g., Velodyne VLP-16 Interface), a spherical imaging module806 (e.g., PointGrey Ladybug 5), a battery module 807 and an inverter808, which are connected with one another via a variety of communicationinterfaces or cables, such as a USB 2.0, a USB 3.0, or an Ethernet port,as shown. The Mapper 800 can be installed on a vehicle that is beingdriven around in some areas to collect high resolution images includingpictures and videos.

In some instances, the antenna module 801 can be an L1/L2 GNSS antenna,offering combined GPS+GLONASS signal reception. In some cases, users canuse the same antenna for GPS only or GPS+GLONASS applications toincrease integration flexibility and reduce equipment costs. The antennaunit 801 can be connected to the GPS/GNSS module 802 via a coaxial cableand transmit the received signals thereto. The GPS/GNSS module 802 cancomprise a receiver for receiving the location signals from the antennaunit 801 and compute position and velocity of the vehicle with highaccuracy. The GPS/GNSS module 802 can also be embedded with an InertialMeasurement Unit (IMU) to further improve the accuracy of themeasurements. To this end, the GPS/GNSS module may additionally compriseone or more accelerometers (e.g., three), one or more gyroscopes (e.g.,three), thereby allowing the receiver to operate in environments wherevery high dynamic and frequent interruption of signals can be expected.

In some instances, the LiDAR module 804 can be configured to performlaser scanning, for example, by using one or more (e.g., 16)laser/detector pairs mounted in a compact housing, which can rapidlyspin to scan the surrounding environment. For example, the LiDAR module804 can be mounted on the rear of a vehicle with the cable pointingdownward and tilted 90 degrees in order to scan the road.

In some cases, the lasers fire thousands of times per second, therebyproviding a rich, 3D point cloud in real-time. Further, the LiDAR module804 can be configured to perform advanced digital signal processing andwaveform analysis to provide high accuracy, extended distance sensing,and calibrated reflectivity data. In some instances, the LiDAR module804 is capable of horizontal Field of View (FOV) of 360°, adjustablerotational speed of 5-20 rotations per second, vertical FOV of 30°, andreturns of up to 100 meters dependent on application. The LiDAR module804 can also synchronize its data with precision, GPS-supplied timepulses, enabling the users to determine the exact firing time of eachlaser.

In some instances, the LiDAR module 804 can be connected to the LiDARinterface module 805, which may serve as an interface box for the LiDARmodule 804. As shown, the LiDAR interface module 805 can receive aonce-a-second synchronization pulse (PPS) issued by the GPS/GNSS module802 and forward it to the LiDAR module 804. Thereby, uponsynchronization, the LiDAR module 804 can set its time stamp to thenumber of microsecond past the hour per coordinated universal time (UTC)time. Thereby, it would be easy to geo-reference the LiDAR data into apoint cloud.

In some instances, the spherical imaging module 806 can receive atrigger pulse from the GPS/GNSS module 802 and capture location-basedvisualizations for display in geographical mapping applications. Forexample, the user can use the spherical vision camera to capture videoin a mobile environment, and view the transmission of images as they arecaptured using the data collection module 803, which can be embodied asa laptop or desktop computer. In some cases, the spherical imagingmodule 806 is capable of 360° video streaming and covering 90% of thevisual sphere. Further, the spherical imaging module 806 can bepre-calibrated and come with its own Software Development Kit (SDK),thereby allowing creation of dedicated applications for imaging capture.

In some instances, the GPS/GNSS module 802, the LiDAR interface module805 and the spherical imaging module 806 each can connect to the datacollection module 803 for data exchange, storage, synchronization andcontrol using various interfaces, such as USB or Ethernet interfaces.The data collection module 803 herein can be a laptop computer or adesktop computer with mass storage, which may serve as a database forstorage of image data.

In some instances, in order to power up one or more of the abovemodules, the battery module 807 is provided as a power supply. Thebattery module 807 herein may be an auxiliary battery on the vehicle,which is able to provide a 12V direct current (DC). As shown, this 12VDC can be directly applied to the GPS/GNSS module 802, the LiDARinterface module 805 and the spherical imaging module 806. However, inorder to properly power the data collection module 803, the inverter 808can be introduced to convert 12V DC into 120V alternating current (AC).

In some instances, a wheel encoder can be used for converting theangular position or motion of a shaft or axle of the vehicle to ananalog or digital code. The analog or digital code can be furtherprocessed into information such as speed, distance and position, therebyproviding further details of the vehicle in motion.

Swarm Processing

Referring to FIG. 10 , in a particular embodiment, once image data isuploaded to the Swarm system as shown in FIG. 1 , a processing pipeline900 will start to process it. All images data is stored in file storageor database S3 and other kinds of data are stored in a database PG. Alldata is associated to edges roughly equal to street segments. In someinstances, as shown in the upper part of FIG. 11 , the pipeline 900 ofthe Swarm system may perform the following steps at corresponding blockswhile exchanging data with associated databases:

-   -   a) Extract frames from video using FFMPEG and associate frames        with metadata at block 901;    -   b) Correct location data using heuristics and mapping data at        block 902 (OSRM+Tiger);    -   c) Locate objects, such as pedestrians, cars, street signs,        street lights, etc., within imagery using trained machine        learning models, such as YOLO detection at block 903;    -   d) If necessary, blur images of pedestrians using OPENCV at        block 904;    -   e) Extrapolate raw pedestrian counts into foot traffic data at        block 905; and    -   f) Detects changes in mapping inventory, including, for example,        new high-priority events, the removal of old high-priority        events, changes in vehicle or pedestrian traffic, and/or so        forth.        Extrapolating Pedestrian Counts

Known methods of estimating pedestrian counts typically involveexamining video of fixed cameras. When using fixed cameras, a fairlyaccurate count of pedestrians can be determined for particularlocations, but little to no pedestrian information is available forareas for which camera coverage does not exist. Using mobile cameras,such as Swarm capture devices described above, can increase the area forwhich pedestrian counts can be determined with a given number ofcameras, but introduces several new challenges. In particular, mobilecameras coverage area and coverage gaps can change in unpredictablemanners and video depicting a particular street segment may be receivedonly on an irregular basis. The following method describes atechnique(s) that can be used to extrapolate pedestrian counts based oninconsistent coverage:

-   -   e1) Swarm images can be queried and congregated into single time        slots that represent a single vehicle pass on a street segment.        This can be performed on a periodic basis, such as daily,        weekly, etc. Similarly stated, video data from multiple Swarm        capture devices can be received and bucketized. For example,        video for each Swarm capture device can be subdivided into a        bucket for each street segment (e.g., intersection to        intersection) captured by that Swarm capture device.    -   e2) Swarm images (e.g., bucketized Swarm images) can be        processed by Darknet YOLO or any other suitable image processing        technique to attribute tags to images. For example, each still        image from a video segment (e.g., corresponding to a bucket        associated with a street segment) can be analyzed to identify        the number of pedestrians in that image. The total number of        people seen for the video segment can be divided by the number        of frames in that video to determine an average pedestrian count        for the video segment, which may be associated with a street        segment.    -   e3) As discussed in further detail below, to determine a        foot-traffic count for a selected street segment, pedestrian        data for a time period and for the selected street segment can        be combined with pedestrian data from immediately adjacent        street segments. Similarly stated, video of any street segments        that intersect the selected street segment can be analyzed to        determine the number of pedestrians for those adjacent street        segments. Video of adjacent street segments can be analyzed over        any suitable time period, such as a week, a month, a year, etc.    -   e4) If multiple videos have been collected for the selected        street segment or adjacent street segments, pedestrian counts        can be de-duplicated by any suitable means (such as aggregating        passes and dividing by number of passes).    -   e5) An hourly histogram (or a histogram of any other suitable        interval) can be calculated based on historic observations of        pedestrians, such as past observations made by Swarm devices        and/or based on publicly available pedestrian count data. Hourly        histogram(s) can be used to extrapolate the pedestrian count        from individual vehicle passes to a total day count. Hourly        histograms can be used to estimate pedestrian counts for the        selected or adjacent street segments during periods (e.g.,        hourly periods) for which no video is available. Daily        projections for the selected street segment and/or adjacent        street segment can be determined by summing hourly histograms        over a 24 hour period.    -   e6) An inter quartile range algorithm can be applied to the        approximate day projections to remove outliers.    -   e7) If the multiple vehicle passes exist for the adjacent street        segments, those adjacent street segment counts can be used to        estimate pedestrian for the selected street segment. For        example, an average (or other measure of central tendency)        pedestrian count of adjacent street segment values can be        calculated, outliers can be removed, and an average (or other        measure of central tendency) between adjacent street segments        and the selected segment can be computed.    -   e8) If selected street segment counts are missing (e.g., if no        video of the selected street segment is available to calculate        pedestrian counts for a particular period), the average (or        other measure of central tendency) of the adjacent street        segments can be used to estimate the average (or other measure        of central tendency) pedestrian count for the selected street        segment and the period. For example, for each adjacent street        segment for which video was received during the time period,        image recognition of such video can be performed to produce a        pedestrian count for that adjacent street segment. An average        (or other measure of central tendency) of pedestrian counts of        adjacent street segments can be used to estimate the pedestrian        count of the selected street segment during that time period.    -   e9) The daily projection of foot traffic for a selected segment        can be used to approximate a monthly foot traffic projection by        multiplying by 365/12.        Detecting Mapping Inventory Changes

Mapping inventory for continuously updated based on newly receivedinformation (e.g., from Swarm system elements and/or Mapper systemelements). For example, permanent and/or temporary changes to roadsand/or traffic conditions can be detected and integrated into availablemaps. In some instances mapping inventory can be tracked for particularlocations, such as points of interest, intersections, addresses, and thelike. In such an instance, video collected from in the vicinity of theparticular location (e.g. an intersection) can be analyzed based, forexample, on metadata associated with the video. Images likely to depictthe particular location (e.g., images likely to be showing crossingthough an intersection) can be analyzed. For each image, the set ofmapping objects that is most commonly detected can be identified. Forthe winning set of objects, the object types and relative positions canbe associated with the intersection. Changes to the particular locationscan be identified by identifying changes in the objects associated withthe intersection. For example, during periodic mapping inventory updates(e.g., nightly, weekly, monthly, etc.) images of the particular locationreceived during that period can be analyzed. If the set of imagesreceived during the period includes objects that match the set ofobjects associated with the intersection, the intersection can beconsidered unchanged. If, however, the set of images received during theperiod does not include objects associated with the intersection, forexample, if objects received during the period includes objects thatdeviate from objects associated with the intersection by more than athreshold amount, then the images, video, and/or associated metadata canbe passed to an Events Service UI, such as shown and discussed abovewith reference to FIGS. 8 and 9 . A GIS analyst and/or machine learningprocess can determine if the inventory has changed and/or update themapping inventory (e.g., indicate that a lane is closed due toconstruction, changes in traffic patterns, lane markings, and so forth).

Further referring to FIG. 12 , in a particular embodiment, once imagedata is uploaded to the Mapper system as shown in FIG. 1 , a processingpipeline 906 will start to process it. All images data is stored in filestorage or database S3 and other kinds of data are stored in a databasePG. The data herein may include raw network PCAP files, PGR files, andGPS files. All data is associated to edges roughly equal to streetsegments. In some instances, the pipeline 906 of the Mapper system mayperform the following steps at corresponding blocks while exchangingdata with associated databases:

-   -   a) Automate Novatel Inertial Explorer to process GPS files into        GNSS files that have corrected location data using the IMU data,        wheel encoder data, and public base station data at block 907;    -   b) Extract LiDAR data from PCAP and combine with GNSS data to        create point cloud slices in CSV files and position/orientation        information in POSE files, and further extract rectified and        panoramic imagery from Ladybug PGR files into JPG files at block        908;    -   c) Combine SCAN, POSE and rectified JPG files to create        registered and colorized point cloud slices at block 909,        wherein the Iterative Closest Point (ICP) is applied herein to        correct for poor GPS data, and Colorized point clouds and        updated POSE information are saved to CSV and POSE files; and    -   d) Perform feature extraction (described below) to identify        streets, sidewalks, poles, and building facades at block 911.        Feature Extraction

In some instances, the feature extraction can be performed in differentstages as follows:

Stage 1

-   -   a) Compute normals at each 3D point by local plane fitting;    -   b) Use the 3D points and normals of the scan perform a RANSAC        plane fitting algorithm, detect planes with vertical normals        (i.e. horizontal planes), keep the biggest two such horizontal        planes and refer the set of points in these horizontal planes as        Points_Ransac;    -   c) Perform a Euclidean clustering (e.g., PCL Euclidean cluster        extraction) on the points in Points_Ransac;    -   d) Keep the largest of the clusters computed in step (c) and        refer the set of these points as Points_Clustered;    -   e) For each 3D point in the scan, compute its maximum distance        from the horizontal planes computed in step (b), wherein the        distance is an approximation of the height from the ground for        each 3D point;    -   f) Add points that are not in Points_Clustered, with a height        from the ground below a given threshold, in Points_Clustered;    -   g) Label each 3D point with a label “ground” if it belongs to        Points_Clustered and a label “ransac” if it belongs to        Points_Ransac and label each 3D point not in Points_Clustered        with a label “non-ground,” wherein each point is assigned a        distance from the ground.

Stage 2

-   -   a) Build a K-D tree index of points in Points_Clustered (i.e.        ground points) for fast search.    -   b) For each 3D point P in Points_Clustered:        -   i) Reconstruct a scanline neighborhood around P, by            computing a neighborhood of points using Euclidean            clustering, and by sorting the points based on the angle of            the sensing direction;        -   ii) Move forward and backward in the scanline neighborhood            to detect sudden changes in depth using a cumulative            statistic. If such a change is detected, the point of change            is marked as a possible curb point; and    -   c) Output: points in scan that could be on curbs are labeled as        possible-curb-point.

Stage 3

-   -   a) Place all possible-curb-points from all scans into a set        referred to as Points_Possible_Curbs;    -   b) Place all ground points from all scans into a set referred to        as Points_Ground;    -   c) Place the sequence of scanner positions (from global poses)        into a list Scanner_Positions;    -   d) Cluster the points Points_Possible_Curbs into components        using Euclidean clustering, and discard small clusters;    -   e) For each cluster from step (d), order points in sequence        following scanner's movement by finding closest scanner        positions in Scanner_Positions;    -   f) Subsample ordered points of step (e) and smooth them, thereby        generating a set of polylines or curves;    -   g) Organize the curves of step (f) into left and right curves        based on whether each curve is to the left or the right of the        moving scanner (represented by Scanner_Positions);    -   h) Process the left curves in sequence, and decide whether to        connect with the next curve. The connection is decided based on        various criteria, such as: 1) whether the connection between the        curves traverses empty space wrt Points_Ground (i.e. in the case        a parked car is occluding the curb), 2) the length of the        connection, 3) the angle between the line connecting the curve        endpoints and the curves themselves, and 4) the distance between        the connecting curve and the scanner trajectory.    -   i) Perform step h) for the right curves; and    -   j) Output: a sequence of curb curves on the left and right of        the sensor trajectory.

Stage 4

-   -   a) Re-label as road the Points that are labeled ground or        non-ground (but at a height from the ground smaller than a        threshold) and being in between a left and right curb curve;    -   b) Label as sidewalk the points that are labeled ground, or        non-ground (but at a height from the ground smaller than a        threshold) that are to the left of the left curb curve;    -   c) Label as sidewalk the points that are labeled ground, or        non-ground (but at a height from the ground smaller than a        threshold) that are to the right of the right curb curve;    -   d) Perform a RANSAC plane clustering to the non-road and        non-sidewalk points into non-horizontal planes, wherein the        points on such RANSAC planes that are to the left or to the        right of the curb curves are labeled as facades.    -   e) Output: points of each Velodyne scan are labeled either as a        road, a sidewalk, facade, or others.

The above has described in detail the operations of the featureextraction using multiple stages. It is to be understood that theprocessing stages herein are only for illustrative purposes and a personskilled in the art can envisage other stages or steps to perform thefeature extraction based on the teaching herein. In addition, the Mappersystem can also process the GPS data at block 910, and performPedestrian detection and blur faces at blocks 912 and 913 using the sameor similar algorithms as the Swarm system.

As previously noted, according to the embodiments of the presentdisclosure, the data from the Mapper and Swarm systems can be merged,thereby improving the efficacy of each system. For example, the systemcan compare the features in Mapper data to features in Swarm data toimprove the Swarm accuracy. The exemplary steps are as follows:

-   -   a) Perform feature extraction on Mapper 2D data to detect        landmarks;    -   b) Perform feature extraction on Swarm data to detect landmarks;    -   c) Compare Swarm features within 100 m of each Mapper feature to        find the same landmarks; and    -   d) Update the Swarm location data when matching features are        found.

Referring to FIG. 13 , in a particular embodiment, a web application,which can be used by a fleet manager and referred to as “fleet managersoftware,” is capable of controlling and monitoring the vehicles usingthe street level intelligence system. The application can be configuredto allow the fleet manager to carry out multiple actions, including butnot limited to:

-   -   a) see the current location of each fleet vehicle;    -   b) download video already offloaded;    -   c) request priority offload over LTE; and    -   d) view driver quality data from Zendrive.

As illustrated in FIG. 13 , there are six vehicles on the road and shownin real-time on the map. The “Vehicle” column lists these six vehicleswith their respective name, traveled distances, and grades. The vehiclenamed “Grimlock” is shown grey in the list as it is offline and inmaintenance. The vehicles herein may be outfitted with the Swarm systemsas discussed above and elsewhere in the specification. It should beunderstood that the user interface shown in FIG. 12 can be a homepage ofthe fleet manager software.

Referring to FIG. 16 , in a particular embodiment, a web application,which can be used by a customer and referred to as “street levelintelligence software,” is capable of providing site intelligencepayload to the customer. The web application herein can be configured toallow the customers to carry out the actions, including but not limitedto:

-   -   a) view a map overlaid with segments for which pertinent data        has been collected;    -   b) select one or more segments and add them to a cart;    -   c) pay for access to segments using a credit card;    -   d) download a payload for those segments containing either 3D/2D        site intelligence or pedestrian analytics;    -   e) access previously-paid payloads; and    -   f) share payloads with partners or other customers.

In some instances, the site intelligence payload may comprise thefollowing:

-   -   a) stitched color 3D point cloud (see Stitching Process as        below);    -   b) panoramic images;    -   c) panoramic viewing software; and    -   d) metadata and instructions.

As an example, the steps for stitching point cloud data are as follows:

-   -   a) Colorized scans are gathered for segment runs, ordered by        capture date;    -   b) Scans are filtered to eliminate stalled traffic redundancy        (movement checked);    -   c) Ensure that the scans have an acceptable count based on        segment length and cleaned scan count;    -   d) Concatenate the filtered scans into a single xyz file; and    -   e) Resultant file is passed to a las converter binary which        exports a las file.

In some instances, the pedestrian analytics payload may contain thefollowing information:

-   -   a) a PNG map of the requested segments overlaid with a        infographics representing estimated pedestrian foot traffic for        each segment; and    -   b) a CSV containing raw data that can be imported into other        software.

As illustrated in FIG. 16 , the customer can input areas or segments ofinterest in the Search field. Further, the customer can select the carticon as shown and place the segments selected from the map into it. Insome cases, the customer can also see the discounts shown in the cart.Finally, the customer can select coin icon as shown to pay the searchfees using for example, credit cards. In response, the street levelintelligence software according to the embodiments of the presentdisclosure can provide items, such as Search miles, Search groups, orSearch cost for customer's viewing. Additionally, the customer can orderother data that is interested to him or her, such as the PNG map of thesearched segments or CSV files.

Non-Transitory Computer Readable Storage Medium

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include one or more non-transitory computer readablestorage media encoded with a program including instructions executableby the operating system of an optionally networked digital processingdevice. In further embodiments, a computer readable storage medium is atangible component of a digital processing device. In still furtherembodiments, a computer readable storage medium is optionally removablefrom a digital processing device. In some embodiments, a computerreadable storage medium includes, by way of non-limiting examples,CD-ROMs, DVDs, flash memory devices, solid state memory, magnetic diskdrives, magnetic tape drives, optical disk drives, cloud computingsystems and services, and the like. In some cases, the program andinstructions are permanently, substantially permanently,semi-permanently, or non-transitorily encoded on the media.

Executable Instructions

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include processor executable instructions, or use ofthe same. A computer program includes a sequence of instructions,executable in the digital processing device's CPU, written to perform aspecified task. Computer readable instructions may be implemented asprogram modules, such as functions, objects, Application ProgrammingInterfaces (APIs), data structures, and the like, that performparticular tasks or implement particular abstract data types. In lightof the disclosure provided herein, those of skill in the art willrecognize that a computer program may be written in various versions ofvarious languages.

The functionality of the computer readable instructions may be combinedor distributed as desired in various environments. In some embodiments,a computer program comprises one sequence of instructions. In someembodiments, a computer program comprises a plurality of sequences ofinstructions. In some embodiments, a computer program is provided fromone location. In other embodiments, a computer program is provided froma plurality of locations. In various embodiments, a computer programincludes one or more software modules. In various embodiments, acomputer program includes, in part or in whole, one or more webapplications, one or more mobile applications, one or more standaloneapplications, one or more web browser plug-ins, extensions, add-ins, oradd-ons, or combinations thereof.

Software Modules

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include software, server, and/or database modules, oruse of the same. In view of the disclosure provided herein, softwaremodules are created by techniques known to those of skill in the artusing machines, software, and languages known to the art. The softwaremodules disclosed herein are implemented in a multitude of ways. Invarious embodiments, a software module comprises a file, a section ofcode, a programming object, a programming structure, or combinationsthereof. In further various embodiments, a software module comprises aplurality of files, a plurality of sections of code, a plurality ofprogramming objects, a plurality of programming structures, orcombinations thereof. In various embodiments, the one or more softwaremodules comprise, by way of non-limiting examples, a web application, amobile application, and a standalone application. In some embodiments,software modules are in one computer program or application. In otherembodiments, software modules are in more than one computer program orapplication. In some embodiments, software modules are hosted on onemachine. In other embodiments, software modules are hosted on more thanone machine. In further embodiments, software modules are hosted oncloud computing platforms. In some embodiments, software modules arehosted on one or more machines in one location. In other embodiments,software modules are hosted on one or more machines in more than onelocation.

Databases

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include one or more databases, or use of the same. Inview of the disclosure provided herein, those of skill in the art willrecognize that many databases are suitable for storage and retrieval ofvehicle, location, image, feature, and street level intelligenceinformation. In various embodiments, suitable databases include, by wayof non-limiting examples, relational databases, non-relationaldatabases, object oriented databases, object databases,entity-relationship model databases, associative databases, and XMLdatabases. Further non-limiting examples include SQL, PostgreSQL, MySQL,Oracle, DB2, and Sybase. In some embodiments, a database isinternet-based. In further embodiments, a database is web-based. Instill further embodiments, a database is cloud computing-based. In otherembodiments, a database is based on one or more local computer storagedevices.

What is claimed is:
 1. A method, comprising: receiving video depictingstreet segments from vehicle-mounted smartphones; bucketizing the videointo buckets that are separately associated with segments of the videocapturing a street segment from the street segments by one of thevehicle-mounted smartphones; identifying (i) a selected segment of thestreet segments for which no video is received during a time intervaland (ii) adjacent segments that are adjacent to the selected segment;calculating a number of pedestrians for each of the adjacent segmentsduring the time interval using image recognition and pedestrian countsfor the adjacent segments by dividing the number of pedestrians for eachof the adjacent segments by a number of video frames associated with theadjacent segments; and estimating a pedestrian count for the selectedsegment according to an average of the pedestrian counts associated withthe adjacent segments.
 2. The method of claim 1, wherein video depictinga particular street segment from the street segments is receivedirregularly.
 3. The method of claim 1, wherein estimating the pedestriancount includes generating histograms of the pedestrian counts observedfor the adjacent segments on an hourly basis.
 4. A system, comprising: aprocessor; and a memory storing instructions that, when executed by theprocessor, cause the processor to: receive video depicting streetsegments from vehicle-mounted smartphones; bucketize the video intobuckets that are separately associated with segments of the videocapturing a street segment from the street segments by one of thevehicle-mounted smartphones; identify (i) a selected segment of thestreet segments for which no video is received during a time intervaland (ii) adjacent segments that are adjacent to the selected segment;calculate a number of pedestrians for each of the adjacent segmentsduring the time interval using image recognition and pedestrian countsfor the adjacent segments by dividing the number of pedestrians for eachof the adjacent segments by a number of video frames associated with theadjacent segments; and estimate a pedestrian count for the selectedsegment according to an average of the pedestrian counts associated withthe adjacent segments.
 5. A non-transitory computer-readable mediumstoring instructions that, when execute by an electronic processor,cause the electronic processor to: receive video depicting streetsegments from vehicle-mounted smartphones; bucketize the video intobuckets that are separately associated with segments of the videocapturing a street segment from the street segments by one of thevehicle-mounted smartphones; identify (i) a selected segment of thestreet segments for which no video is received during a time intervaland (ii) adjacent segments that are adjacent to the selected segment;calculate a number of pedestrians for each of the adjacent segmentsduring the time interval using image recognition and pedestrian countsfor the adjacent segments by dividing the number of pedestrians for eachof the adjacent segments by a number of video frames associated with theadjacent segments; and estimate a pedestrian count for the selectedsegment according to an average of the pedestrian counts associated withthe adjacent segments.